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CONFIDENTIAL Designator 


apiVersion: redhat/v2 
kind: PrincipalConsultant 
metadata: 

name: Giovanni Marigi 
namespace: ITI 
websites: 

linkedin: giovannimarigi 

github: hifly81 
annotations: 

specialist: openshift, cloud, integration, kafka, bpm 
labels: 

sports: mountain bike, trekking 
spec: 
replicas: 1 
containers: 

- image: redhat.io/giovanni:latest 
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CONFIDENTIAL Designator 


apiVersion: redhat/vl 

kind PrincipalSoftwareEngineer 

metadata: 

name Paolo Patierno 

namespace Red Hat, Messaging & loT team 
annotations: 
cncf/maintainer: Strimzi 
eclipse/committer: Vert.x, Hono & Paho 
microsoft/mvp: Azure 
labels: 

family: dad of two, husband of one 
sports: running, swimming, motogp, vr46, ssc napoli 
spec: 
replicas 1 
containers: 

- image patiernohub.io/paolo:latest 
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Running Apache Kafka on OpenShift 

The challenges 


• Apache Kafka is *stateful* which means we require ... 

o ... a stable broker identity 

o ... a way for the brokers to discover each other on the network 
o ... durable broker state (i.e., the messages) 
o ... the ability to recover broker state after a failure 

• StatefulSets, PersistentVolumeClaims, Services, CRDs 

• CRDs have some advantages as well, flexible data structure, 
possibility to set permissions for the CRD resources 



AMQ Streams on OpenShift 


• Simplify the way for deploying and handling Apache Kafka on 
OpenShift 

• Follows the Kubernetes “operator” model 

• OpenShift 3.11,4.x 

• Based on OSS project called Strimzi 
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AMQ Streams on OpenShift 

What is Strimzi ? 


• Open source project licensed under Apache License 2.0 

• Sandbox project under the Cloud Native Computing Foundation (CNCF) 

• Focuses on running Apache Kafka on Kubernetes and OpenShift: 

o Container images for Apache Kafka and Apache Zookeeper 

o Operators for managing and configuring Kafka clusters, topics or users and much more 

• Provides Kubernetes-native experience for running Kafka on Kubernetes and OpenShift 

o Kafka cluster, topic and user (and more) as Kubernetes custom resources 


0 Web site: http://strimzi.io/ 
o GitHub: https://aithub.com/strimzi 
Twitter: @strimziio 
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Operators within the AMQ Streams architecture 
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AMQ Streams Custom Resources 


: kafka.strimzi.io/vlbetal 
: KafkaUser 

metadata : 

: my-user 

labels : 

: my-cluster 

spec : 

authentication : 

type: tls 

authorization : 

: simple 

acls : 

- resource: 

type: topic 
nan : my-topic 


: kafka.strimzi.io/vlbetal 
: KafkaTopic 

metadata : 

nan : my-topic 

labels : 

r: my-cluster 

spec : 

partitions: 1 
replicas: 1 


: kafka.strimzi.io/vlbetal 
kind: Kafka 

metadata: 

: my-cluster 

spec : 
kafka: 

version: 2.1.1 
replicas: 3 
listeners : 
plain: {} 
tls: {} 
config : 

offsets.topic.replication.factor : 3 



KAFKA CLUSTER 

How to deploy 
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KAFKA CLUSTER 

How to update 
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AMQ Streams 

Features and roadmap 


• Kafka Mirror Maker 2.0 

o Better support for active-active scenarios 
o Consumers offset mapping 

• Kafka Connect Connectors 

o Deploy your Kafka Connect connector with a custom resource 
o No need to use the Kafka Connect REST API 

• Kafka Exporter 

o More metrics about consumers and lag 

• OAuth with Keycloak support 



AMQ Streams 

Features and roadmap 


• JMXtrans 

o Reading JMX metrics from the cluster and write to different 
destinations (i.e. InfluxDB, OpenTSDB,...) 

• Cruise Control... coming in the next release! 

o Cluster rebalancing (based on CPU, disk, network I/O,...) 
o Kafka Cluster Rebalance CRD to define the rebalance “goals 



Specific use cases 

Why should you use Strimzi/AMQ Streams ? 

• Messaging 

o “Classic messaging” 
o Event store / Event sourcing 
o Microservices asynchronous communications 

• Data management 

o DB/Mainframe Offloading 
o Data Replication (CDC) 

• Tracking/Metrics 

o Website Activity tracking (original linkedin use case: page views, searches,...) 
o aggregating stats from distributed applications -> centralized feeds of operational data 

• Log Aggregation 

o Collect logs from multiple services 
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Why should you use Strimzi/AMQ Streams ? 


“The more complex your deployment is, the 
more value operators can provide to you” 
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Customer’s use case 

The challenge 

• Process a huge amount of data related to log’s aggregation and analysis use case 

• A new Kafka cluster that processes 95% of messages related to applications and 
Openshift infrastructure 

• Obtain a better support in terms of design the infrastructure, manage the 
infrastructure, automate the infrastructure 

• Run the solution on top of Kubernetes that is identified as a reference 
infrastructure for the applications. 
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Customer’s use case 

Volumes 


System 

Calculated avg size of 
messages 

Calculated messages/sec 
(approx) 

Calculated throughput 
(approx) 

App cluster #1 

4KB 

XXX 

XXX 

Openshift cluster 

30KB 

XXX 

XXX 

Total 

8KB (weighted avg) 

XXX 

65 MB/s 


System 

Size (approx) per week 

App cluster #1 

XXX 

Openshift cluster 

XXX 

Total 

70 TB 
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Customer’s use case 


Sizing 


Component 

# of instances 

CPU (per inst.) 

Zookeeper 

5 

4 vCPU 

Kafka broker 

12 

12 vCPU 

Cluster operator 

1 

2 vCPU 

Entity operator 

1 

4 vCPU 


RAM (per inst.) 

Disk (per inst.) 

16 GB 

500 GB 

128 GB 

12TB 

1 GB 

n/a 

4GB 

n/a 


























Customer’s use case 

Kubernetes Quotas 

• For kafka resources: 

o The container requests 8 CPU. 
o The container requests 96Gi memory, 
o The container limits 10 CPU. 
o The container limits 128Gi memory. 

• For zookeeper resources: 

o The container requests 2 CPU. 
o The container requests 8Gi memory, 
o The container limits 4 CPU. 
o The container limits 16Gi memory. 
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First *very simplified Architecture 
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Thank you 

Red Hat is the world’s leading provider of enterprise 
open source software solutions. Award-winning 
support, training, and consulting services make 
Red Hat a trusted adviser to the Fortune 500. 
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